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PROGRAM EVALUATION: AN INTRODUCTION 


The purpose of this paper is to establish a common ground of 
understanding and a basic framework for discussing the subject of 
program evaluation. An earlier version of this paper was prepared as 
background material for consideration of several evaluation initiatives. 
This material can also serve as background material for two documents 
currently being prepared by the Office of the Comptroller General: "A 
Guide on the Program Evaluation Function in Federal Departments and 
Agencies" and "Principles for the Evaluation of Programs by Federal 
Departments and Agencies". 


This discussion paper begins by describing what program 
evaluation entails (Section 1) and outlines the evolution of evaluation 
in the federal government (Section 2). It describes the benefits of 
program evaluation in Section 3 and gives examples of recent evaluation 
studies of government programs (Section 4). This is followed in 
Section 5 by a discussion of the main factors to be considered in 
planning an appropriate evaluation study. Section 6 discusses the 
limitation of conducting and using evaluation studies. The paper 
concludes (Section 7) with a list of characteristics of successful 
evaluation studies. 


The paper discusses these topics, for the most part, in general 
terms and is not limited to describing program evaluation as it is carried 


out inside the federal government. Where appropriate, specific approaches 
to program evaluation are referred to, such as in the case of the 


Treasury Board policy on evaluating programs or evaluation studies 
that could be done for central agencies, Cabinet or Parliament. 


The footnotes to this paper supply further information and 
references on this challenging and complex subject. 


1. | WHAT IS PROGRAM EVALUATION? 


There is no agreement in the literature on what program 
evaluation should entail. The terminology used varies widely and is 
itself the cause of much confusion. Simply stated, program evaluation 
is the formal assessment of the continued relevance and of the effec- 
tiveness of existing programs. That is, it involves the systematic 
gathering of verifiable information on a program and demonstrable 
evidence on its results. The Office of the Comptroller General, in 
discussing government program evaluation, has expanded on this: 


Program evaluation is the periodic, independent and objective 
review and assessment of a program to determine in light of 
present circumstance, the adequacy of its objectives, its design 
and its results both intended and unintended. Evaluations will 
call into question the very existence of the program. Matters 
such as the rationale for the program, its impact on the public, 
and its cost effectiveness as compared with alternative means of 
program delivery are reviewed. 


In this context a program is a group of government activities, 
powers or functions that can be related in a logical fashion to a 
given objective or set of objectives. A (program) evaluation study is 
an analysis of specified aspects of a program and its effectiveness 
and an evaluation report is a report of the study, its findings and 
conclusions. Program evaluation, as a management function, is the 
evaluation of programs with respect to all relevant evaluation issues 
(see Table 2). Much of this paper focuses on evaluation studies 
rather than on the function of program evaluation. An expansion on 
these definitions is provided later (Section 5) after a number of 
additional concepts have been introduced. 


2. DEVELOPMENT OF PROGRAM EVALUATION IN THE FEDERAL GOVERNMENT 
The review and evaluation of existing programs has always 


been a part of managing in government. The nature and content of the 
reviews and evaluations have evolved from focusing mainly on the 


resources used by programs (the dollars spent and numbers of people 
employed) to examining how the resources are used, the purposes of 
programs and their impacts and effects on society. 


This change of emphasis has been necessary. The government 
has been called upon over the past three decades to provide an increas- 
ing array of goods and services to the public. Many current government 
programs reflect significant and sophisticated attempts to improve 
social and economic conditions in an increasingly complex society. As 
government activities have grown, the objectives and results of such 
programs have come under increased public scrutiny. 


The increased concern with the purposes and hence the results 
of government expenditures can be traced to the 1960 Royal Commission 
on Government Organization (Glassco). Prior to Glassco, the analysis 
of programs relied mainly on the annual audits of government expendi- 
tures. A  post-audit was performed for Parliament by the Auditor 
General and a pre-audit was conducted by the Comptroller of the Treasury. 
These audits ensured the probity of expenditures (the program inputs) 
but did not concentrate on the purposes and results of programs. The 
Glassco Commission's reports, published in 1962 and 1963, led to a 
general decentralization of financial and management control and to 
the development of more modern management techniques. The traditional 
concern for audit was coupled with a new concern about whether a 
program was conducted in a manner consistent with its legislated 
intent and specific operating goals. 


To assist in the reorientation of departmental managers to 
assume more responsibility for managing their affairs and to provide a 
better insight into the primary purposes of departmental operations, a 
new financial management approach - PPB, (Planning, Programming, 
Budgeting) - was introduced in 1968. "With its emphasis on defining 
program objectives and showing the full costs for each program''>, PPB 


was intended to introduce planning and programming to the budgetary 
process. Its forward-looking multi-year basis was to provide a frame- 
work to facilitate the Treasury Board's allocation of resources. * In 
practice, difficulties were encountered in trying to make the system 
operational and PPB did not live up to all its expectations. 


However, PPB did contribute to the new emphasis on examining 
the intended results of programs. Since 1970-71, this has been reflected 
in the federal government's annual Estimates book which cross-classified 
expenditures by agency, by program, and by objects of expenditure. 


During the 1970s there was an increase in evaluative activity 
throughout the government across the spectrum of program forecast and 
review, operational auditing, to in-depth, formal, quantitative evalua- 
tions of the effectiveness of specific programs. These latter formal 
evaluations were carried out on a highly selective basis normally in 
response to major policy planning priorities or issues rather than as 
part of the ongoing examination of existing programs by departmental 
management. 


Thus while management in government has always carried out 
evaluation studies, formal, systematic, and periodic evaluation of 
programs was initiated in 1977 with the issuing of the Treasury Board 
Policy Circular 1977-47, "Evaluation of Programs by Departments and 
Agencies". To a considerable extent this policy was built upon the 
best practices already in existence in a number of departments and 
agencies. The general statement of the Policy is that: 


Departments and agencies of the federal government will periodi- 
cally review their programs to evaluate their effectiveness in 
meeting their objectives and the efficiency with which they are 
being administered. 


The focus of the Policy is on the evaluation of programs, as 
opposed to, in particular, the evaluation of systems and procedures 
which are, in general, examined in internal audit. > As envisaged in 
the policy, program evaluation is considered to cover a number of 
basic program evaluation issues: 


Program Rationale: Does the program make sense? 


Impacts and Effects: What has happened as a result of 
the program? 


Objectives Achievement: Has the program performed as 
expected? 


Alternatives: Are there better ways of achieving the 
results? 


The purpose of program evaluation in the federal government 
undertaken pursuant to this policy is to ensure that deputy heads of 
departments and agencies have the appropriate information on _ the 
results of their programs in order to be able: 


to account for the public monies for which they are respon- 
Sible; 


to make informed decisions on the management and resourcing 
of their programs; and 


to provide quality advise to Ministers. 


Changes in organizational structure have accompanied the 
evolution of the program evaluation function in the federal government. 
The Office of the Comptroller General was created in June 1978. This 
office has broad authority and responsibility for administrative 
practices and controls in the areas of financial management and proce- 
dures for program evaluation in the Government of Canada. More specif- 
ically, the Office has the responsibility to see that departments and 
agencies have established program evaluation procedures conforming to 
Treasury Board policy. 


3. | BENEFITS FROM PROGRAM EVALUATION 


There is a growing acceptance that periodic program evaluation 
is a key function in the management of government departments. As has 


been observed? 


Evaluation is necessary because of the scarcity of resources. 
. evaluation involves the allocation of limited resources among 
competing activities. 


The wider implementation of program evaluation will lead to 
a better understanding of the operations of programs and of the actual 
results of programs. This will contribute to: 


improving accountability for government programs ; 


enhancing the ability of the government to allocate scarce 
resources for the greatest effects; 


increasing the capability of the government to plan ahead 
and thereby to devise better programs to meet policy goals; 


clarifying a program's contribution to policy initiatives; 
and 


improving the management and administration of government 
programs, and thereby providing better government for 
Canadians. 


These are the direct consequences. More and better evalua- 
tions will also have two more general consequences: 


the regular involvement in the evaluation of programs will 
encourage a program's staff to examine critically their own 
assumptions and behaviour and thereby to experiment and to 
find ways of performing better; and 


the acceptance of program evaluation by concerned citizens 
and groups as one legitimate and useful tool in the process 
of determining a program's worth to society, should help to 
pee misunderstanding and disagreement over a program's 
effects. 


4. SOME EXAMPLES OF EVALUATION STUDIES OF GOVERNMENT PROGRAMS 


The diversity of types of evaluation studies carried out on 
federal programs is illustrated in Table 1. Evaluation studies are 
done by different people and for different reasons. For example, it 
is evident that none of the studies listed in Table 1 were undertaken 
pursuant to the 1977 Treasury Board policy, largely because of timing 
but also in several cases because the clients of the studies were not 
deputy heads. It is also clear that different evaluation studies use 
different techniques. The types of studies in the table range from 
determining the impacts on society to questioning the specific compo- 
nents or activities comprising the program. Evaluation techniques 
range from econometric modelling to interviewing beneficiaries of the 
program and experts in the subject-matter area. 


This wide range of possible types of evaluation studies 
emphasizes the need for careful refinement of the term "program evalua- 
tion". It also implies that care must be taken to ensure that for any 
evaluation study an appropriate evaluation strategy is developed, one 
that will address the right issues and satisfy the client of the 
study. 


5. | PLANNING APPROPRIATE EVALUATION STUDIES 


Planning for an evaluation study is perhaps the most critical 
phase of any evaluation. Evaluation studies can be costly in terms of 
both financial and human resources and experience has shown that many 
such studies in the past have turned out not to be useful (see 
Section 6.4). At least part of the reason for non-use of evaluation 
studies in the past has been a lack of adequate consideration before 
commencing the study as to what was needed, what could be done, and 
what would be done. Such planning to determine the proper focus and 
methodology - here called an evaluation assessment study - is essential 
for the design of an appropriate evaluation study.’ 


sisAleue 
DLuyawoU0dA 


squedtoiqued 
jo sAaauns al dwes 


SOLPLALZIC 
quawyuedsep 
jo sisAjeuy 


sisAleue 
DLUyYsWoUodZ 


SLPLILJJO pue squedxe 


$O SMOLAASYUT 


ABo LopoyzewW 


UOLZENLeAG 


UOLY 
-edioiqued adu0j unoge|, uo szijoueq 
JIN $0 3D9ZJe BYZ eULWUazep OL 


azepuew s ,weuboud 
ay2 JO 3xXa}U0D ayy UL wesBboud 
YY JO SALYLALZIS BY MALABU O] 


SOALJELZLUL MaU YUsWa|dwL 
O27 pauinbau Sadunosed 94 3Seq 94N} 
-lpuadxa szL ULYQLM Wous AJ LquapL OL 


szueub gey Jo 
SSOUBALPIOJJO BYY BULWUazap OL 


Sassauyeam 

pue syjzbuauzs s,weuboud ayy ssasse 
0} pue Sartzoefgo SzL ZoowW 0} 
Ayitiqe s,wesuboud ay} aeqluosep oO] 


UOLzeNLeAW 
30 asodund 


nn EE EEE y EEE SE=IEEyE=nnPSnPa TINS SS=EPS=a nna 


SWYYDONd LNSWNYSAOD JO SSIGNLS NOILWATWAZ AWOS 


| Lounog 
2LWOU0DA 


goueUul4 

pue eLueyIUIES 
pueog Aunseaul 
‘uoLyzeuBb Lwwy 
pue yUuswAo| dwy 


yeLueysuses pueog 
Aansead] pue 340 


aoUaWWO) pue 
apeu, ‘Auysnpu] 
JO JUSWZUed|q 


9942 LWWo) 
aDueULJ aZeUaS 


qUeLLI 


L eLgel 


LL6L 


8Z6L 


8ZL/LL6L 


9Z6L 


weuboud 

S}lLjaueg uoLssiw 
-wWoj) adueunsUy 
quawAo| dweun 


weuboug HuLuLteul 
Jamoduey epeue) 


(340) PUSWUOULAUG 
Y Salusysl4 
40 YUaWZUedsEG 


weuBboug sjzueuy 
dALJUIDUT 
quswdo|aaeaq 
pue youeasay 


weuBboud 


UOLYEPOWWOIIY S}AOM 
DLLGNg JO JUsWyUeda|g 


_...._ Se. 3 2 6 fe ST 


payen[ erg 
weuboud 


Oe le ll. lS FT 


An appropriate evaluation study of a program requires consid- 
eration of: 


the nature of the program to be evaluated; 

the questions to be addressed; 

the client of the evaluation study; 

the evaluation method to be adopted; 

ie resources and time available for carrying out the study; 
an 


the need to ensure credibility. 


These six factors will assist those responsible for carrying 
out the evaluation study and the client to develop an appropriate 
evaluation strategy. The six factors are discussed in order below. 


5ok The Nature of the Program to be Evaluated 


The first factor to be considered in designing an evaluation 
study is a definition or description of what is to be evaluated. What 
constitutes a "program" for evaluation purposes is not always obvious. ® 
A program may or may not coincide with the organizational structure of 
government or of a department. In particular, a program for evaluation 
purposes may not be a "program as described in the government's 
annual Estimates. Within the federal government, considerable work 
has gone on to develop suitable program components for evaluation 
purposes pursuant to the Treasury Board policy on evaluation, i.e., 
groups of activities having a common objective established at the 
level of concern of the deputy head, and of appropriate size or impact 
to be a focus of decision making on programming at the departmental 
level. (In this discussion paper, the term program rather than program 
component is retained to describe what is evaluated.) The best dis- 
aggregation into program components suitable for evaluation purposes 
is often not obvious until after the department's activities are 
examined in some detail. 
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Having identified the activities which will constitute the 
program to be evaluated, it is necessary to have some understanding of 
how the program is supposed to work: an understanding of its basic 
rationale. That is, how the program activities produce program outputs 
which in turn result in particular impacts and effects on society. 
Objectives for the program can be specified in terms of the program 
outputs, the program's intended impacts and effects, or both, depending 
on the nature of the program and the level of objective being defined. 


The state of development of the program must also be consid- 
ered. Most programs cannot be evaluated in a meaningful way until 
they have been operating for some time. Furthermore, the type of 
service or good provided by the program may also influence the evalua- 
tion design, '° as will the type of intervention being used: expendi- 
ture, regulatory or tax expenditure. 


5w2 The Questions to be Addressed 


In evaluating a program a great number of questions could be 
asked and a wide array of information about the program's operations 
and results could be collected and analyzed. (As the examples in 
Table 1 illustrate, these could range from broad questions on the 
impacts of the program to more limited questions about the acquisition 
of resources for the program.) Basic to designing an appropriate 
study is a clear understanding of precisely which questions are to be 
addressed. This provides the necessary focus for the study. 


Table 2, based on the four general evaluation issues mentioned 
earlier in Section 2, lists seven basic evaluation questions which can 
be addressed in a program evaluation. The table in fact provides an 
operational definition of the term "program evaluation", as it is 
being used in the federal government. 


Tab 
BASIC PROGRAM E 


Classes of Evaluation Issues 


Program Rationale 


Impacts and Effects 


Objectives Achievement 


Alternatives 


Program rationale issues 
the program in light of present 


government policy. 
questioned by asking 


Here the very existence of the program 
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le 2 
VALUATION ISSUES 


Basic Evaluation Questions 


To what extent are the objectives 
and mandate of the program stil] 
relevant? 


Are the activities and outputs of 
the program consistent with its 
mandate and plausibly linked to 
the attainment of the objectives 
and the intended impacts and 
effects? 


What impacts and effects, both 
intended and unintended, resulted 
from carrying out the program? 


In what manner and to what extent 
does the program complement, dupli- 
cate, overlap or work at cross- 
purpose with other programs? 


In what manner and to what extent 
were appropriate program objectives 
achieved as a result of the program? 


Are there more cost-effective alter- 
native programs which might achieve 
the objectives and intended impacts 
and effects? 


Are there more cost-effective ways 
of delivering the existing program? 


focus on the continued relevance 
social 
ieee ee) 


of 


and economic conditions and 


be 


if the program is still needed, even assuming it 


-12- 


is producing as expected. A good understanding of the rationale of 
the program should be developed, by comparing the current program 
activities with the mandated activities and examining the continued 
plausibility of the link between the program's outputs and both its 
objectives and other intended impacts and effects. 


Consideration of the impacts and effects of a program implies 
a broad focus, an attempt to determine what has happened as a result 
of the program. It is concerned with the immediate impacts of the 
program and its more widespread, diffuse, or longer term effects, both 
intended and - often of more interest - unintended, regardless of the 
stated or claimed objectives of the program. This includes distribu- 
tional conditions, as well as impacts and effects on and by related 
programs. 


Objectives achievement issues are concerned with determining 
the manner and the extent to which appropriate objectives are achieved 
as a result of the program. This focus has often been narrowly taken 
as the only focus for program evaluation. 


Finally, the consideration of alternatives is a yardstick 
for assessing the relative worth of the program. The objectives may 
have been met and there be no negative impacts or effects, but there 
may be better ways of achieving the objectives or intended impacts and 
effects: achieving the same results for less cost, obtaining more or 
better results for the same cost or achieving proportionately more or 
better results with increased costs. The program would then be more 
effective. Alternatives could include, as appropriate, both alterna- 
tive ways of delivering the program and alternative programs to achieve 
the objectives and intended impacts and effects. These issues can be 
approached by considering questions of economies of scale for the 
program as well as the effect of discontinuing the program. 
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Some or all of these various issues can serve as the focus 
of a particular evaluation study. All should be addressed in a compre- 
hensive program evaluation study. It may not, of course, be always 
desirable or indeed practical to provide in-depth answers to each 
evaluation question. For example, program evaluation in government 
carried out under the Treasury Board policy calls for each question to 
be seriously considered for analysis and investigation, and an appro- 
priate evaluation study to be designed, taking into account the needs 
of the client (the deputy head), the priority of the evaluation issues, 
the expected benefits in each case, and the resources available. 
Conversely, evaluation studies called for by Cabinet or Parliament 
would likely focus only on the specific issues of interest to Cabinet 
or Parliament. 


Answers to the questions in Table 2 indicate the kind of 
information to be obtained in an evaluation study. As a result, terms 
of reference for a particular study should be as specific as possible 
as to what exactly is required, and not be couched in general terms as 
in the table. For example, objectives for a service-type program 
would be specific as to the level of service aimed at - the quantity, 


quality, timeliness, and fairness of the service. |! 


Of course, 
depending on the study design, information may not be collected on all 
questions: a study may not collect data on how well objectives have 
been achieved if the main focus of interest is on determining the 
extent and nature of a particular negative unintended effect. What 
needs to be determined for a particular evaluation study is the type 
of information that is required to ensure that, in the eyes of the 
client, the study addresses the right questions. This is the main 
purpose of the evaluation assessment study, done before the _ formal 


study begins. 


The type of evaluation issue being considered may also 
affect the definition of a program for evaluation purposes. An exami- 
nation of the rationale of a program would clearly focus on the parti- 
cular program while an assessment of the impacts on a client group may 
very well be expanded to include all programs affecting the group. 
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5.3 The Client of the Evaluation Study 


In addition to knowing the questions to be addressed and 
hence the desired focus of the evaluation study, it is critical that 
the client of the study be identified and the relevant interests be 
understood. There may also be numerous users of any study and the 
clients of the study may wish to take the users' concerns into account. 


There are a large number of potential clients (and users) of 
evaluation studies of government programs: 


Cabinet Committees 

media 

Parliament 

program planners 

public 

program managers 
researchers 

senior government managers. 


Given the wide diversity of interests and roles of these 
different clients, an evaluation study of a particular program done 
for different clients will usually require a different study design, 
will answer somewhat different questions and will produce different 
types of findings. 22 


For example, the 1977 evaluation study of the Unemployment 
Insurance Commission Benefits (Table 1) was done by the Economic 
Council of Canada. The Council was interested in determining the 
large-scale effects of changes in the program. The study was aimed at 
achieving a better understanding of the program so that future program 
planning could be improved. An evaluation study of the same program 
by Parliament, on the other hand, might be aimed instead at determining 
the effects on individuals or groups of recipients of UIC benefits, or 
at determining the impact of the program on small communities. Another 
client might be interested in the impact of the program as an income 
supplement program rather than as an insurance program, or in whether 


- 15 - 


the same intended effects could be achieved better through the use of 
a negative income tax scheme. A program manager, on the other hand, 
might be interested in small-scale effects that could be improved 
through incremental changes in the program rather than through major 
program changes. 


Program evaluation studies conducted pursuant to the Treasury 
Board Policy Circular 1977-47 (see Section 4.2) have as their client 
the deputy head of the appropriate department or agency and while, as 
mentioned above, such evaluation would entail addressing all questions 
in Table 2, the particular evaluation studies done would reflect the 
specific interests and concerns of the deputy head. This would be 
evident in the form of the questions addressed and, quite probably, in 
the groups of activities focused on for each study. Parliament, the 
Cabinet or central agencies, as a client for evaluation studies, may 
very well be interested in different issues and in examining different 
groupings of activities. It could be expected, for example, that 
evaluation for Parliament, the Cabinet or central agencies would 
involve proportionally more interdepartmental studies than would 
program evaluation done for individual deputy heads. 


5.4 The Evaluation Method 


The fourth aspect to be considered in the development of an 
appropriate evaluation strategy is the evaluation method or technique 
to be used. The diversity both of types of evaluations and of clients 
for evaluation studies, requires the use of different evaluation 
methods, from quantitative analyses to qualitative investigations, 
from rigorous to Jess rigorous assessments, as is evident from 
Table hor. An evaluation study which seeks to assess the impact of a 
particular government program may have to use sophisticated analytical 
techniques. For example, for planners, computer simulation techniques 
may be required to analyse the impact of regional economic incentives 
on the Atlantic provinces. On the other hand, an evaluation done for 


management in departments-* 


- 16- 


may be quite flexible as to methodology. Its objectives are 
pertinence, accuracy, and credibility with reference to one or 
more program management issues. Its audience is principally made 
up of program managers who are not highly sensitive to the social 
science researcher's demand for experimental rigor, or to the 
auditor's demand for standards of evidence. 


It should be recognized that for certain programs it may be 
impossible to obtain definitive quantitative answers to some evaluation 
questions. This does not. imply that there is no feasible evaluation 
method. Less rigorous quantitative approaches or qualitative approaches 
are almost always available to provide useful insights into the program 
and its effects and to give at least partial answers to the more 
difficult evaluation questions. 


55 The Resources and Time Available 


In designing an appropriate evaluation study, it must be 
recognized that there are limited resources and time available to 
carry out such a study. These constraints can often best be addressed 
during an evaluation assessment phase. 


Usually, the evaluation study can be designed to meet speci- 
fied resource and time constraints. This may require a_ trade-off 
between the quantity and the quality of the information to be obtained. 
For example, the findings of an evaluation to determine the effective- 
ness of a manpower training program would be much less ambiguous if a 
special-purpose control group were created. This would be more expen- 
sive and take much longer than the use of a comparison group derived 
from existing administrative data. The 1977 evaluation study of the 
Canada Manpower Training Program (see Table 1) was faced with this 
trade-off and chose the comparison-group approach, which gave useful 
but less definitive findings. 


- 17 - 


Evaluation studies will be more or less costly depending on 
the depth of inquiry into specific issues, the number of issues 
addressed and the availability of information. For much evaluation 
work - especially when the impacts of the program are being investi- 
gated - information from outside the program is required. Obtaining 
this can be costly and time-consuming but may be essential to a useful 
study. !° 


Evaluation studies done for different clients usually entail 
different resource and time requirements and must incorporate appro- 
priate time constraints. Evaluations done for management often have a 
short-term perspective and must be finished in time for decisions on 
program management. Otherwise, no matter how technically good the 
study, it may be too late to be usable and hence inappropriate. 


Similarly, evaluations done for Parliament or central agen- 
cies must take into account the time required for the study to be 
carried out. Evaluations may be asked for when a particular issue is 
in the public's mind and must be completed in a time acceptable to the 


citent.!® 


Evaluations done for researchers and planners often require 
more time because of their concentration on the impacts of programs. 
They also tend to make use of more complex analytical techniques and 
of more specialized expertise. !/ 


Finally, evaluation studies should be cost-justified. While 
there is no general rule to determine if an evaluation study will be 
worth its cost, some attempt can usually be made to see whether the 
likely benefits from the study are anything near the cost. An evalua- 
tion study is not justifiable when possible benefits are judged clearly 
to be less than the costs that would be incurred. 
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5.6 The Need to Ensure Credibility 


Probably the most important requirement of an evaluation 
study is that it be credible. Impartiality, accuracy, and under- 
standing must be maintained in an evaluation. | The degree to which 
these are achieved will differ for different studies. Each evaluation 
study must be credible in the eyes of the client in order for the 
findings to be seriously considered. 


In the context of evaluations done for deputy heads pursuant 
to the Treasury Board policy on evaluation, credibility of the evalua- 
tion will be achieved through a number of mechanisms. Deputy heads 
determine what is evaluated and how, and are an integral part of the 
evaluation process. The evaluation units in departments are organiza- 
tionally independent from program management, giving them the freedom 
to be more objective. The Office of the Comptroller General plays a 
role in assuring objectivity and accuracy of evaluation studies by 
reviewing selected studies. Finally, public review of any evaluation 
reports, especially if Access of Information legislation is passed, 
will help to increase the credibility of the reports in the eyes of 
the public. 


For program managers to be satisfied with evaluations of 
their programs, they have to be familiar with the strengths and weak- 
nesses of the evaluation techniques used. In addition it has to be 
made evident to them that the evaluators have dealt with the practi- 
calities of their programs in the study. '? 


5 oh Summary 


The examples in Table 1 and elsewhere in the text illustrate 
the wide range of possible kinds of issues that can be addressed under 
the label "program evaluation". The need for appropriate evaluation 
studies is evident. In particular, the client of the study and the 
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evaluation questions addressed determine the evaluation strategy that 
should be adopted and hence the focus, emphasis and ultimately the 
findings of the study. Designing appropriate evaluation studies 
requires careful planning before the actual study commences: an 
evaluation assessment is almost always required. No single evaluation 
strategy is always better than another. The use of highly sophisti- 
cated evaluation techniques does not guarantee a more useful evaluation. 
Indeed, the reverse has sometimes been the case. What is needed is an 
appropriate study: one designed for the particular purpose in mind. 
Only in this way can non-productive and often costly program evaluation 
efforts be reduced to a minimum. 


As a result, the terms of reference for each evaluation 
study should be carefully developed and include both the _ specific 
questions that are to be addressed and an outline of the methods to be 
used. This will be of crucial importance, to ensure that the clients 
of the study - deputy heads, Cabinet or Parliament - receive evalua- 
tions which they find useful and suited to their needs. “2 


6. LIMITATIONS AND REALISTIC EXPECTATIONS FOR EVALUATION STUDIES 


The many benefits that result from more and better evalua- 
tions of government programs were described earlier in Section 3. 
However, as indicated by Canadian and foreign experience, there are 
important limitations to the conduct and use of evaluation studies. 
This section outlines some of the main lessons learned in this evolving 
area. 


Many governments carry out evaluations of their programs. 
The status and scope of program evaluation in any country is determined 
by that country's constitutional arrangements and the way in which it 
has managed government operations in its particular political setting. 
Many countries and institutions within each country have tried various 
approaches to evaluation with varying degrees of success. All have 
found evaluation of effectiveness difficult to implement in a useful 
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way. For example, in Sweden, a lot of evaluation activity takes place 
in its agencies and through Royal Commissions. At the centre, however, 
the National Audit Bureau now concentrates its efforts on systems 
audits for economy and efficiency rather than on effectiveness evalua- 
tions, in an attempt to improve management practices. The British 
Treasury in its Program Analysis Reviews conducts evaluations which 
are used occasionally for planning purposes and have little apparent 
impact on improved management. In the U.S., the General Accounting 
Office's stated goal is to conduct what it calls "program results 
reviews" of all programs. In practice, it finds that program ratio- 
nales are often not sufficiently developed for such review. Neverthe- 
less, much program evaluation is carried on in the U.S. in the various 


government departments and agencies at both federal and state level.2! 


Certain points emerge from the Canadian and foreign program 
evaluation experience to date. Expectations for program evaluation 
are tempered by the following problems: 


Program Objectives: the difficulty of specifying measurable 
objectives for all programs and creating valid measures of 
objective achievement. 


Methodology: the lack of standard, successful methods for 
the study of social and economic problems. 


Resource Constraints: limitations on all categories of 
resources for the conduct of evaluations. 


Using Evaluation Findings: difficulties connected with the 


adoption of and taking action on findings by both managers 
and policy makers. 


A Program's Worth: the lack of a final bottom line means 
that the final judgement is subjective. 


6.1 Program Objectives 


Effectiveness evaluation activity in the past has tended to 
concentrate on the analysis of the extent to which program goals and 
objectives are achieved@* (Table 2). Success with this type of study 
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has been limited. The evaluator is often unable to do his or her job 
because of the ill-defined nature of many program objectives, and the 
program manager often resents attempts to quantify and measure what he 
or she sees as inherently qualitative and unmeasurable. 


The objectives of some public sector programs are simply not 
measurable (and may never be). In Sweden, the National Audit Bureau 
has explicitly shifted from an exclusive goals approach, to an assess- 
ment of processes and procedures, working co-operatively with managers. 
The Auditor General of Sweden reflected recently on 10 years of experi- 
ence in Sweden: 2° 


When effectiveness auditing began in Sweden great weight was 
attached to the matter of goals. The auditors were to utilize 
the goals set up by the politicians as a point of departure for 
an assessment of the degree of goal fulfilment in the activities 
of government agencies. Experience has taught us that often this 
shot simply isn't on the board. One activity will often have 
several goals to aim at and the goals may be hazy and diffuse. 
This situation is not likely to change. Often, politicians are 
unwilling to tie their hands by stating clear and unequivocal 
goals for their policy. As a rule, therefore, we in Sweden have 
had to abstain from any assessment of goal fulfilment.... Indeed, 
we are working on the assumption that in a longer time perspective 
audit of operations will come to predominate effectiveness 
auditing in Sweden. 


The underlying problem is not that some programs lack objec- 
tives. At the policy level every program has objectives although they 
may be expressed rather’ generally. However, specific measurable 
objectives that enable the evaluator to link program objectives and 
policy objectives in a logical manner are often difficult to define. 2+ 
The extreme difficulty of objectives-oriented evaluation was recognized 
by the Report of the Independent Review Committee on the Office of the 
Auditor General of Canada (the Wilson Committee) which reported in 


1975:2° 


Present problems in measuring the effectiveness of spending 
programs revolve around two major questions. First, how does one 
define in quantifiable terms what Parliament expects to achieve 
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in enacting new legislation or by approving a new program through 
voting funds in an appropriation act? Many programs defy quanti- 
fication. For example, how can one translate the purposes of the 
Department of External Affairs into quantitative terms that 
permit objective measurement of performance? While other programs 
are susceptible to some form of measurement, they may be intended 
to achieve a number of different objectives. Some are measurable 
(for example, the reduction of unemployment or improvement in the 
standard of living) and some are intangible (such as overcoming a 
feeling of injustice). Even if some of these identification 
problems can be resolved, there remains a second major question: 
How can the results achieved be measured? In some cases, present 
statistical information may provide a reasonable answer; but in 
others, new statistical techniques will have to be developed to 
provide a dependable measure of results. 


Even where measurability is not a problem, it can be very 
difficult to develop agreement on what measures correctly represent 
the stated objectives: ©° 


Indicators may emerge from the political-bureaucratic process 
which minimize problems or make them more amenable to solutions 
and which suit the emotional commitments and interests of the 
decision-makers and those who influence them. Indicators may be 
selected to promote the adoption of desired policies, to make 
these policies appear as successful as possible, and to direct 
public discussion in the desired directions. 


Again, assuming that measurability is not a problem, accep- 


tance of programs may depend on different groups perceiving the objec- 


tives of the program to be different: 


Legislation requires ambiguity in the statement of goals so that 
coalitions can be formed in support of it, and each group can 
believe that the legislation serves its own special purposes. 
Insisting that the government's purposes be clear may reduce the 
scope of legislation and thus limit government involvement in 
social programs. 


Also, an emphasis on measurement will turn attention to 
those things which can readily be measured. The consequent flow of 
attention and resources to the area may be at the expense of greater 
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real need elsewhere. What is measurable may not be of most interest. 
Furthermore, over emphasis on measurable indicators may turn the 
indicators into the program's objectives. 78 


There are a number of other reasons why a program's objec- 
tives may be deliberately left vague or unstated, as opposed to being 
impossible or difficult to define in principle. Objectives may be 
left unspecified for reasons relating, for example, to security or 
unity within the nation. In other cases the program may be aimed at 
providing some service for which there is very little known about how 
to produce the service effectively, even while the program is earnestly 
desired. Examples are programs designed to promote racial or religious 
tolerance or to limit some undesirable behaviour. Objectives are left 
vague to permit flexibility in the implementation and management of 
the program, and to provide for responses to new demands or changes in 
the program environment. Resources can be moved to activities which 
appear to work as more is learned. Finally, governments of all modern 
democratic countries pay for some programs, usually for moral or 
ethical reasons, which might appear to many taxpayers to be ineffective. 
Treatment for some types of mental disorders is a case: any treatment 
whatever is thought to be better than no treatment. Withholding 
treatment indicates hopelessness to the patient and seems callous on 
the part of the professionals involved. Highlighting the fact that 
treatment cannot be reasonably demonstrated to be effective may not be 
a useful contribution to the problem. But the general social objec- 
tives may be to indicate concern for such patients and to continue 
research on this problem, and therefore objective statements will be 
vague. 


Having indicated that for some programs (usually social) a 
focus on objectives achievement may not be appropriate, two points 
must be made clear. First, for many other programs, efforts devoted 
to clarifying and measuring progress towards program objectives will 
be a very useful evaluation focus. Not doing objectives achievement 
analysis should be the exception, not the rule. Secondly, even if 
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focusing on objectives is not appropriate in a particular case, this 
in no way should limit addressing other evaluation issues. In parti- 
cular, in such cases, it becomes essential to determine just what are 
the results of the program. 


6.2 Methodology 


A common problem and an important limitation to the evalua- 
tion of a program's effectiveness is the difficulty of isolating the 
effects or impact of the program from those of other programs or other 


factors.2> 


At any one time our society is subject to a large number 
of influences. For technical or ethical reasons, few of these can be 
adequately controlled in order that the effects of the program under 
study can be unambiguously determined. The difficulty can apply to 
programs which appear to be quite limited in_ scope. Iq 20955, 
Connecticut instituted a crackdown on speeding. During the year there 
were 284 traffic deaths compared to 324 in the previous year. Was the 
increased enforcement responsible for this effect? It may have been 
in part, or in whole; but, because of many other influences such as 
weather conditions, the use of seat belts or other safety features it 
would be most difficult to determine the exact contribution of each 


factor to the reduced death rates? 


The problem of uncontrolled outside factors manifests itself 
in other ways. Some government programs involve making grants or 
contributions to other organizations who then carry out their own 
project or program alone. Some organizations may refuse to accept 
funds if one of the conditions for doing so is the provision of infor- 
mation on their clients for program evaluation purposes. Such a 
dilemma requires a subjective judgement on whether to make the grant 
or contribution. If the judgement is that a grant ought to be made in 
any case, then the success or failure of the grant or contribution 
program will not be able to be evaluated by the government using 
information on the client population. 
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When the program being evaluated is composed of smaller 
components or projects it may not make sense to aggregate the findings 
of separate reports. Furthermore, at the level of the larger program, 
the kind of effectiveness questions asked may be quite different.°! 
For example, for a project-based program the effectiveness of each 
project can be investigated: what are the effects produced and to 
what extent are objectives achieved? However, the effectiveness of 
the program as a whole may relate to determining what would have 
happened without the program. How many projects would have gone on in 
any event? This is a different kind of evaluation than that of summing 
up the findings of evaluations of the individual projects. The success 
of aggregating a number of smaller evaluation studies will vary with 
each situation. 


The dynamic nature of many programs further frustrates the 
attempt to conduct in-depth evaluations of programs. The single 
program or the program environment may not remain stable long enough 


to allow a definitive evaluation study to be carried out. 24 


Information systems which have been developed for admin- 
istrative purposes may not be suitable for in-depth evaluation analysis. 
Historical information may not exist. In addition to the often high 
cost of obtaining information on the program, data may not be available 
for reasons of protection of the privacy of citizens - no matter how 
otherwise desirable it may Mee For example, the use of personal 
income tax and personal medical records for the evaluation of many 
programs may be strongly questioned by citizens concerned with the 
right to privacy. 


6.3 Resource Constraints 


The conduct of evaluation studies is often costly and time- 
consuming. Adequate resources may not be available to conduct a 
credible study. In such cases either the resources must be increased 
or the study curtailed or postponed. 
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Program evaluation is a developing area and is not a well- 
established body of formal knowledge. This imposes another kind of 
resource constraint. There is a general lack of relevant program 
evaluation experience in government. The increased application of 
program evaluation will require training for both managers and 
practitioners. 


A more complex constraint lies in_ the multi-discipl inary 
nature of some evaluation studies. Evaluation activities draw from 
many fields and disciplines such as accounting, economics, education, 
industrial engineering, the management sciences, political science, 
medicine, psychology, sociology, statistics and urban affairs.°" 
Because each program or problem will demand a somewhat different 
combination of skills, an evaluator may be out of his/her depth in 
some aspect of the study and the quality of the evaluation could 
suffer. >> For example, most economists or auditors charged with 
evaluating the New Horizons program for people retired from the labour 
force could miss or misinterpret information concerning changes in 
attitudes and behaviour. Such information might be more properly 
collected and interpreted by other types of social scientists. 


6.4 Using Evaluation Findings 


The uses of evaluation findings were outlined earlier in 
Section 3. A particular evaluation study may, of course, be either 
more useful or less useful to its client. Although different clients 
will be using evaluation results with different purposes in mind, for 
all clients and potential users of evaluation studies there are limi- 
tations to using the findings. 


Part of the problem is that evaluation studies conducted for 
one client usually are not well suited for use by others; they do not 
address the relevant questions. For example, evaluations designed for 
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policy planners will usually deal in comparative analysis, merely 
touching on individual program areas of importance to managers, and 
hence may be of little use for program managers. °° 


Often, evaluation studies are used by persons other than the 
original client. >! This is inevitable, but the study should make it 
clear what issues are and are not addressed. Of course, if a study 
has not addressed the issues of relevance to the client it is unlikely 
to be of much use. 


Users should also realize that evaluation studies may not 
produce uncontested findings and may not unequivocally lead to clear 
recommendations. Particularly in the area of social programs, practi- 
tioners acknowledge the difficulty of obtaining results from effec- 
tiveness evaluation studies which clearly recommend one feasible 
course of action. Summary findings tend to be ambiguous to the layman 
and controversial to the expert. Applied social science research is 
sometimes not compatible with political or management needs for infor- 
mation on government programs. 


In addition, since evaluation studies focus on the impacts 
of programs, certain findings may have political implications which 
have not been addressed in the study. In politically sensitive areas 
this could limit the effective implementation of some recommendations. 
Related is the fact that the client of the study may not have complete 
control] over taking action on the findings and recommendations. To 
change a government program often involves several actors who may not 
agree on the interpretation of the findings or indeed on the focus of 
the study: from their viewpoint it may have missed key issues. 
Deputy heads, for example, might be unable to make effective use of 
the results of an evaluation study if program changes were to have 
interdepartmental impacts. 
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Evaluation findings and conclusions will sometimes not be 
acted upon as recommended for different reasons. There are fairly 
serious limits on how much, and how rapid, change a program can and 
should tolerate. Managers must be able to count on some financial 
commitment to their programs, especially those which represent signi- 
ficant interventions in human lives. Whether or not. the program 
results are those that were intended by the program, they may be very 
important in lives of individuals. It can also be difficult to obtain 
competent staff to continue to work in programs where shifts are rapid 
and unexpected; people develop loyalties to programs and those who 
have a choice may take their talents out of the Public Service. 
Consequently, politicians and managers alike may often prefer to shift 
activities and work for broad continuity and incremental innovation, 
even though an evaluation report suggests that a more radical change 
would be more effective. 


The usefulness of evaluation studies can be greatly increased 
if the findings are written and communicated in straightforward language. 
Too often evaluation studies are written in over-specialized language 
understandable only to the evaluators and a_ limited professional 
community. Clear presentation can help decision-makers better under- 
stand the study and its findings. °° 


6.5 Determining a Program's Worth 


The term "program evaluation" could imply that the evaluation 
Study will determine or comment upon the real value of the program - 
its ultimate worth. This will rarely be the case. Program evaluation 
is simply an aid to decision-makers. Program evaluation cannot and 
should not replace the final judgement of the responsible decision- 
maker. H.S. Havens, the Director of the Program Analysis Division of 
the General Accounting Office in the United States, has stated this 
point succinctly. 2% 
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Evaluations are not black boxes into which funds can be poured 
and optimum decisions come out. Only the political process can 
judge the value of program results and compare their priority to 
the results of other programs competing for a limited budget. 


Most programs have multiple objectives, and there are unavoid- 
able limits to how far the information can be aggregated into summaries 
to explain, in a single bottom line, whether things are going well or 
badly. Douglas Hartle, a former Deputy Secretary of the Treasury 
Board, suggests that this would occur in all cases where programs have 
multiple objectives. For example, if the objectives of a government 
were to increase GNP, reduce unemployment and reduce inflation, it 
would be a simple matter to measure progress toward these individual 
objectives. It would, however, be very difficult to determine any 
general improvement in the total economic condition if one of these 
indicators showed improvement and the other two indicated deteriora- 


tion.” 


This is the job of the appropriate decision-makers, although 
the evaluator may be asked to express his or her opinion and to make 


recommendations. 


Even were a program to have a single objective which it had 
achieved, it does not necessarily follow that the program has _ been 
worth while. The objective may have been inappropriate, the program 
may have created a number of negative unintended impacts and effects 
or it may have been possible to achieve the objective at lower cost. 
The evaluation study should consider all these possibilities, and 
provide the decision-makers with the relevant information so that they 
can judge the worth of the program. 


7. WHAT MAKES AN EVALUATION STUDY SUCCESSFUL? 


Experience gained in program evaluation in Canada and else- 
where should be recognized and taken into account in developing new 
initiatives. This experience suggests that a number of aspects of 
successful evaluations can be identified: 
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The program being evaluated should be carefully identified 
and described. 


An evaluation assessment - a pre-evaluation planning phase - 
should be carried out to determine the appropriate focus and 
methodology for the evaluation study. 


The evaluation study should have specific terms of reference 
which include, among other things, the specific evaluation 
questions to be addressed, so that the particular objective 
of the study is well understood. 


It should be expected that the potential benefits of carrying 
out an evaluation would outweigh the cost. 


Evaluation studies should involve the manager of the program 
being evaluated and, to enhance objectivity, should involve 
independent evaluators. 


The evaluation study should be timely in the sense of being 
completed in time for its consideration in the appropriate 
decision-making context. 


There should be a sufficient mix of resources of the right 
quality available for the particular evaluation study. 


The methodology used should be credible to the client. 


Expectations as to the possible findings of the evaluation 
study should be realistic. 


The language of evaluation reports should not be over- 
specialized. 


The evaluation report should clearly describe what the 
specific questions to be addressed were, how they were 
analyzed and investigated, what evidence was collected and 
what conclusions followed from this evidence. 


The evaluation study should have a specific client who has a 
commitment to examining the findings. 
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